Experiment with Different Values

In the process so far, we’ve used several constants in your calculations:

• Lookback period: How far back we calculate the mean and standard deviation for each status code. The value we used is 60 minutes.
• Entries Threshold: The least amount of entries we want to get an alert for. The value we used is 10.
• Z-Score Threshold: The z-score after which we classify the value as an anomaly. The value we used is 6.

Now that we have a working query to backtest, we can experiment with different values.

This is a chart showing the alerts our system identified in the past 12 hours:

To get a sense of each parameter, let’s adjust the values and see how it affects the number and quality of alerts we get.

If we decrease the value of the z-score threshold from 3 to 1, we should get more alerts. With a lower threshold, more values are likely to be considered an anomaly:

If we increase the entries threshold from 10 to 30, we should get fewer alerts:

If we increase the backtest period from 60 minutes to 360 minutes, we get more alerts:

A good alerting system is a system that produces true alerts at a reasonable time. Using the backtesting query, you can experiment with different values that produce quality alerts you can act on.